What We Talk About When We Talk About Diversity

Novelty and diversity are functions of three things: a system’s performance at ad-hoc retrieval, its ability to order documents diversely, and the collection over which the system is run. Ideally, our diversity evaluation framework (test collections and evaluation measures) would sort systems by their skill at ordering documents. Unfortunately, the current framework as exemplified by TREC is actually dominated by ad-hoc performance and insensitive to document ordering. In this paper, we define a measure of diversity difficulty for a query and subtopics irrespective of any ranked list, and suggest cases for future failure analysis of diversity evaluation frameworks. We conclude with a suggested retrieval task that will sort systems by their ability at ordering documents given a query’s diversity difficulty while controlling for their ability to perform ad-hoc retrieval.